TITLE OF THE INVENTION 

IMAGE PROCESSING APPARATUS AND METHOD 

5 FIELD OF THE INVENTION 

This invention relates to an image processing 
apparatus and method for extracting an object 
contained in a moving image and detecting an abnormal 
10 frame. 

BACKGROUND OF THE INVENTION 

Processing for separating and combining images on 
15 a per-object basis by utilizing digital techniques has 
become of interest in recent years. In particular, 
MPEG- 4 coding has been standardized as an 
international standard in the coding of moving images. 
MPEG- 4 coding makes it possible to perform 
20 coding/decoding object by object and offers the 

possibility of various applications that have been 
difficult to achieve heretofore, examples being an 
improvement in coding efficiency, data distribution 
conforming to the transmission path and re- 
25 manipulation of images. 

A technique referred to as the background 
subtraction method is known generally as a method of 
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extracting an object in the processing of moving 
images. This is a method in which points where 
changes occur are detected by comparing a previously 
captured background image and an actual input image. 
5 The principles of this method will now be described in 
simple terms. 

First, let Pc(x,y) and Pb(x # y) represent a pixel 
value of an input image and a pixel value of a 
background image, respectively, at coordinates (x,y) 
10 in the plane of an image. The difference between 
Pc(x,y) and Pb(x,y) is calculated and the absolute 
value thereof is compared with a certain threshold 
value Th. 

An example of a criterion formula is as follows: 
15 |Pc(x,y) - Pb(x,y)| ^ Th ... (1) 

If the absolute value of the difference in 
Equation (1) is equal to or less than the threshold 
value Th, this indicates that there is no change at 
the point (x,y) and, hence, it is decided that Pc is 
20 background. On the other hand, if the absolute value 
of the difference in Equation (1) is greater than the 
threshold value Th, then this indicates that the value 
has changed and that the point is one that should be 
extracted. By performing the above discrimination 
25 process at all points in an image, extraction of one 
frame is achieved. 

Fig. 13 is a block diagram illustrating the 
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configuration of a conventional system in which the 
background subtraction method and MPEG- 4 coding are 
combined. An image input unit 1201 in Fig. 13 is for 
inputting a moving image and is exemplified by the 
5 image sensing unit of a camera. Since the background 
subtraction method requires a background image for 
reference, the background image is generated by a 
background image generating unit 1202. Generating the 
background image by capturing one frame of an image 
10 beforehand under conditions in which an object does 
not yet appear is the simplest method. 

A background subtraction processor 1203 generates 
shape data representing the shape of an object from 
the input image from the image input unit 1201 and the 
15 reference image from the background image generating 
unit 1202. The image from the image input unit 1201 
and the shape data from the background subtraction 
processor 1203 are input to an encoder 1204 frame by 
frame and the encoder proceeds to apply coding 
20 processing. The encoder 1204 will be described as one 
which executes coding in accordance with the MPEG- 4 

coding scheme. 

If an object is to be coded, it is necessary to 
code the object shape and position information. To 
25 accomplish this, first a rectangular area that 

encompasses the object is set and the coordinates of 
the upper left corner of the rectangle and the size of 
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the rectangular area are coded. The rectangular area 
is referred to as a "bounding box" . The area within 
an object expressed by an image signal and shape 
signal is referred to as a "VOP" (Video Object Plane) . 
5 Fig. 14 is a block diagram illustrating in detail 

the structure of the encoder 1204, which executes VOP 
coding according to the prior art. It should be noted 
that the inputs to the encoder 1204 are image 
luminance and color difference signals as well as the 
10 shape signal. These signals are processed in 
macroblock units. 

First, in an intra-mode, a DCT unit 1301 applies 
a discrete cosine transform (DCT) to each block and a 
quantizer 1302 quantizes the resultant signal. 
15 Quantized DCT coefficients and the quantization width 
are subjected to variable -length coding by a variable- 
length encoder 1312. 

In an inter -mode, on the other hand, a motion 
detector 1307 detects motion by a motion detection 
20 method a primary example of which is block matching 

with respect to another VOP that is adjacent in terms 
of time. A motion-vector prediction unit 1308 detects 
a macroblock that is predicted to exhibit the smallest 
error relative to a macroblock of interest. A signal 
25 indicating motion toward a macroblock that is 

predicted to exhibit the smallest error is a motion 
vector. An image to which reference is made in order 
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to generate the predicted macroblock is referred to as 
a "reference VOP". 

A motion compensator 1306 applies motion 
compensation to the reference VOP based upon the 
5 detected motion vector, thereby acquiring the optimum 
predicted macroblock. Next, the difference between 
the next macroblock of interest and the corresponding 
predicted macroblock is obtained, DCT is applied to 
the resulting difference signal by the DCT unit 1301 

10 and the DCT coefficients are quantized by the 
quantizer 1302. 

The shape data, on the other hand, is coded by a 
shape coding CAE unit 1309. What actually undergoes 
CAE coding here are boundary blocks only. With regard 

15 to a block inside a VOP (all data within the block 

lies within the object) and a block outside a VOP (all 
data within the block lies outside the object), only 
header information is sent to the variable -length 
encoder 1312. A boundary block that undergoes CAE 

20 coding is processed in a manner similar to that of the 
image data. Specifically, in the interframe mode, the 
boundary block undergoes motion detection by the 
motion detector 1307 and motion-vector prediction is 
performed by the motion-vector prediction unit 1308. 

25 CAE coding is applied to the difference value between 
the motion-compensated shape data and the shape data 
of the preceding frame. 
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However, two problems described below arise with 
the background subtraction method set forth above. 

The first problem involves the fact that this 
method presumes that there is no change in the 
background image. Specifically, the problem is that 
if a value in the background changes owing to a change 
in illumination or the like, a stable result of 
extraction will not be obtained. A method of 
detecting a change in the background image by a 
statistical technique and updating the background 
image appropriately has been disclosed in the 
specification of Japanese Patent Application Laid-Open 
No. 7-302328 as a solution for dealing with this 
problem . 

15 The second problem is how to deal with an 

instance in which a flash is fired in the middle of a 
scene or in which one object crosses in front of 
another object. These instances will be described 
with reference to the drawings. Fig. 15 is a diagram 
20 useful in describing shape data representing the shape 
of an object in a case where a flash is fired in the 
middle of a scene according to an example of the prior 
art. Reference numerals 1401 and 1402 denote frame 
data at a certain time and frame data at the next 
25 instant in time, respectively. The scene is 

illuminated by a flash in the second of these frames . 
Reference numeral 1403 denotes frame data at the 
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instant in time that follows the frame data 1402. It 
will be understood that the frame 1402 illuminated by 
the flash differs greatly from the other results of 
extraction (1401, 1403). 
5 An instance where there is a change in background 

illumination, which is the first problem mentioned 
above, primarily is merely a change in luminance 
value. In the case of a flash, however, which is the 
second problem mentioned above, hue also changes. As 

10 a consequence, accurate correction of background is 

difficult to achieve. Further, even if accurate shape 
data of an object has been obtained, the image data of 
the object itself also undergoes a major change. With 
a method such as MPEG- 4, therefore, which uses an 

15 interframe difference, coding efficiency cannot be 
raised and the image appears unnatural visually. 

Pig. 16 is a diagram useful in describing shape 
data representing the shape of an object in a case 
where one object crosses in front of another object in 

20 an example of the prior art. Reference numeral 1501 
in Fig. 16 denotes frame data immediately before one 
object (e.g., a vehicle) crosses in front another 
object (a person), reference numerals 1502 and 1503 
denote frame data when the vehicle is crossing in 

25 front of the object that is the person, and reference 
numeral 1504 denotes frame data immediately after the 
vehicle has crossed in front of the person. If a 
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second object thus happens to be extracted together 
with a first object, there is a major increase in 
amount of information ascribable to a major change in 
the shape data and image data. This leads to a major 
5 decline in image quality. Since this case represents 
a phenomenon that is entirely different from a change 
in background illumination, it is difficult to deal 
with the background image by the updating method. 

10 SUMMARY OF THE INVENTION 

The present invention has been proposed in order 
to solve the problems of the prior art and its object 
is to provide an image processing apparatus and method 

15 whereby even if shape data representing the shape of 
an object in a moving image undergoes a major change 
temporarily, a frame that is to be corrected can be 
detected and corrected in ideal fashion and it is 
possible to execute moving-image coding that is 

20 outstanding both visually and in terms of coding 
efficiency. 

According to the present invention, the foregoing 
object is attained by providing an image processing 
apparatus comprising: moving-image input means for 
25 inputting a moving image composed of a plurality of 
frames; background -image acquisition means for 
acquiring a background image relating to the moving 
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image that has been input; object extraction means for 
extracting an object by comparing each of the frames 
constituting the moving image with the background 
image; abnormal-data discrimination means for 
5 discriminating whether shape data representing the 
shape of the extracted object is abnormal or not; 
shape-data correction means for correcting the shape 
data based upon result of discrimination of the shape 
data; image-data correction means for generating image 

10 data, which represents the image of the object, 

conforming to the shape data that has been corrected 
by the shape-data correction means; and coding means 
for coding the shape data and the image data. 

The image processing apparatus according to the 

15 present invention is such that the abnormal-data 

discrimination means discriminates the necessity of 
correcting the shape data by comparing the shape data 
of the object from frame to frame of a plurality of 
frames that differ in time. 

20 Further, the image processing apparatus according 

to the present invention is such that the abnormal- 
data discrimination means includes: first comparison 
means for comparing the shape data of the object using 
a present frame and a preceding frame; second 

25 comparison means for comparing the shape data of the 
object using the preceding frame and a succeeding 
frame; and decision means for deciding that correction 
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of the present frame is necessary if a difference in 
the shape data of the object between the present frame 
and the preceding frame is large and, moreover, a 
difference in the shape data of the object between the 
5 preceding frame and the succeeding frame is small. 

Furthermore, the image processing apparatus 
according to the present invention is such that the 
abnormal-data discrimination means includes: first 
comparison means for comparing the shape data of the 

10 object using a present frame and a preceding frame; 
second comparison means for comparing the shape data 
of the object using the present frame and a succeeding 
frame; and decision means for deciding that correction 
of the present frame is necessary if a difference in 

15 the shape data of the object between the present frame 
and the preceding frame is large and, moreover, a 
difference in the shape data of the object between the 
present frame and the succeeding frame is large. 
Furthermore, the image processing apparatus 

20 according to the present invention is such that the 
abnormal-data discrimination means includes: first 
comparison means for comparing the shape data of the 
object using a present frame and a preceding frame; 
second comparison means for comparing the shape data 

25 of the object using frames following the present frame 
and the preceding frame; and decision means for 
deciding that correction of prescribed frames from the 
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present frame onward is necessary in a case where a 
difference between the present frame and the preceding 
frame is large and, moreover, a difference in the 
shape data of the object between the preceding frame 
5 and the frames following the present frame is small. 

Further, the image processing apparatus according 
to the present invention is such that the first or 
second comparison means performs the comparison using 
a comparison of any of the area, perimeter, wavelet 

10 identifier, circularity, centroid or moment of the 
shape data, or a combination thereof. 

Further, the image processing apparatus according 
to the present invention is such that the shape-data 
correction means corrects the shape data of the object 

15 in an Nth frame, which has been determined to be 

abnormal, using an (N-l)th or earlier frame determined 
to be normal and an (N+l)th or later frame determined 
to be normal. 

Further, the image processing apparatus according 

20 to the present invention is such that the shape-data 
correction means further includes detection means for 
detecting corresponding points in the shape data 
between an (N-l)th or earlier frame and an (N+l)th or 
later frame. 

25 Further, the image processing apparatus according 

to the present invention is such that the detection 
means obtains amount of motion of the corresponding 
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points by performing pattern matching with respect to 
one or each of a plurality of prescribed areas. 

Further, the image processing apparatus according 
to the present invention is such that the amount of 
5 motion of the corresponding points is amount of 

movement and amount of rotation of the position of the 
corresponding points obtained by an affine 
transformation . 

Further, the image processing apparatus according 
10 to the present invention is such that the shape-data 
correction means replaces shape data in an Nth frame 
determined to be abnormal with shape data of an (N- 
l)th or earlier frame determined to be normal . 

Further, the image processing apparatus according 
15 to the present invention is such that the image-data 
correction means generates image data representing the 
image of the object from a frame different from a 
present frame in terms of time. 

Further, the image processing apparatus according 
20 to the present invention is such that the image-data 
correction means generates an Nth frame determined to 
be abnormal from an (N-l)th frame determined to be 
normal and an (N+l)th frame determined to be normal. 

Further, the image processing apparatus according 
25 to the present invention is such that the image-data 
correction means replaces an Nth frame determined to 
be abnormal with an (N-l)th frame determined to be 
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normal . 

Further, the image processing apparatus according 
to the present invention is such that the coding means 
is coding means compliant with an MPEG- 4 visual coding 
5 scheme . 

Other features and advantages of the present 
invention will be apparent from the following 
description taken in conjunction with the accompanying 
drawings, in which like reference characters designate 
10 the same or similar parts throughout the figures 
thereof . 

BRIEF DESCRIPTION OF THE DRAWINGS 

15 The accompanying drawings, which are incorporated 

in and constitute a part of the specification, 
illustrate an embodiment of the invention and, 
together with the description, serve to explain the 
principles of the invention. 

20 Fig. 1 is a block diagram illustrating the 

structure of an image processing apparatus according 
to an embodiment of the present invention; 

Fig. 2 is a flowchart for describing the 
operation of the image processing apparatus according 

25 to this embodiment; 

Fig. 3 is a flowchart useful in describing 
processing for detecting an abnormal frame, in which a 
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flash has been fired, in an abnormal-data 
discrimination unit acoording to a first embodiment of 
the present invention; 

Fig. 4 is a flowchart useful in describing 
5 another example of discrimination processing for 

detecting a frame, in which a flash has been fired, in 
the abnormal-data discrimination unit according to the 

first embodiment; 

Fig. 5 is a flowchart useful in describing 
10 processing for correcting shape data and image data in 
the first embodiment; 

Fig. 6 is another flowchart relating to 
correction of shape data and image data in a third 
embodiment of the present invention; 
15 Fig. 7 is a diagram useful in describing shape 

data in the first embodiment; 

Fig. 8 is a diagram useful in describing result 
of detecting corresponding points in shape data in the 

first embodiment; 
20 Fig . 9 is a flowchart for describing operation of 

the abnormal-data discrimination unit according to a 
second embodiment of the present invention; 

Fig. 10 is another flowchart useful in describing 
processing for correcting shape data and image data in 
25 the second embodiment; 

Fig. 11 is a diagram useful in describing shape 
data in the second embodiment; 
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Fig. 12 is a diagram useful in describing 
processing for detecting corresponding points in shape 
data in the second embodiment; 

Fig. 13 is a block diagram illustrating the 
5 configuration of a conventional system in which the 
background subtraction method and MPEG- 4 coding are 
combined; 

Fig. 14 is a block diagram illustrating in detail 
the structure of an encoder for performing VOP coding 

10 according to the prior art; 

Fig. 15 is a diagram useful in describing shape 
data representing the shape of an object in a case 
where a flash is fired in the middle of a scene 
according to an example of the prior art; and 

15 Fig. 16 is a diagram useful in describing shape 

data representing the shape of an object in a case 
where one object crosses in front of another object in 
an example of the prior art. 

20 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Preferred embodiments of the present invention 
will now be described in detail with reference to the 
drawings . 

25 Fig. 1 is a block diagram illustrating the 

structure of an image processing apparatus according 
to a preferred embodiment of the present invention. 
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In Fig. 1, an image input unit 101, background 
image generating unit 102, background subtraction 
processor 103 and encoder 104 have functions identical 
with those of the image input unit 1201, background 
5 image generating unit 1202, background subtraction 
processor 1203 and encoder 1204, respectively, 
described above with reference to Fig. 13. Thus, this 
embodiment is characterized by the fact that the 
encoder 104 executes coding processing that is 

10 compliant with the MPEG- 4 visual coding scheme. 

The image processing apparatus according to this 
embodiment further comprises an abnormal -data 
discrimination unit 105 for detecting an abnormality 
by examining shape data from the background 

15 subtraction processor 103; a shape-data correction 

unit 106 for correcting the shape data in a case where 
an abnormality has been detected by the abnormal -data 
discrimination unit 105; and an image -data correction 
unit 107 for correcting image data in accordance with 

20 the shape data corrected by the shape-data correction 
unit 106. The altered image data and shape data is 
input to the encoder 104 and is coded thereby. 

Fig. 2 is a flowchart for describing the 
operation of the image processing apparatus according 

25 to this embodiment. 

As shown in Fig. 2, a moving image is input from 
the image input unit 101 (step S101) and a background 
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image (reference image) is generated by the background 
image generating unit 102 (step S102). The background 
subtraction processor 103 generates shape data 
representing the shape of an object from the input 
5 image that enters from the image input unit 101 and 
the reference image that enters from the background 
image generating unit 102 (step S103). The abnormal- 
data discrimination unit 105 discriminates an abnormal 
frame (step S104) and the shape-data correction unit 

10 106 and image-data correction unit 107 correct the 
shape data of the object and the image data of the 
object, respectively (step S105). The encoder 104 
codes the image data and shape data after the 
correction thereof (step S106). 

15 More specifically, the image processing apparatus 

according to this embodiment acquires a moving image, 
which is composed of a plurality of frames, from the 
image input unit 101, acquires a background image 
relating to the moving image using the background 

20 image generating unit 102, extracts an object by 

comparing the background image with each of the frames 
constituting the moving image using the background 
subtraction processor 103, and determines whether the 
shape data of the extracted object is abnormal using 

25 the abnormal-data discrimination unit 105. If the 
shape data is determined to be abnormal, the shape 
data is corrected by the shape-data correction unit 
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106, the frame is corrected by the image-data 
correction unit 107 using the shape data corrected by 
the shape-data correction unit 106, and the corrected 
shape data and image data is coded by the encoder 104. 
5 Discrimination of an abnormal frame at step SI 04 

and processing for correcting the shape data and image 
data at step S105 will be described in detail in 
application to two embodiments set forth below. It 
should be noted that processing other than that set 
10 forth below is as described above. 
<First Embodiment > 

An image processing apparatus according to a 

first embodiment of the present invention will be 

described with reference to Figs. 3 to 8. This is 
15 processing for dealing with the problem (firing of a 

flash in the middle of a scene) described above with 

reference to Fig. 15. 

Fig. 3 is a flowchart for describing processing 

for detecting an abnormal frame, in which a flash has 
20 been fired, by the abnormal -data discrimination unit 

105 according to the first embodiment. 

First, the abnormal-data discrimination unit 105 

generates shape data of the present frame (step S201). 

This is data that is the result of extraction using a 
25 background subtraction method similar to that of the 

prior art. Next, the shape data of the preceding 

frame is compared with the shape data of the present 
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frame (step S202) . 

Several methods are conceivable as methods of 
comparing the shape data. A comparison of areas is 
one of the simplest examples of comparison. By way of 
5 example, the number of pixels in shape data regarded 
as an object makes up the area of the object. 
Further, it is possible to adopt perimeter as a 
parameter, and expression by a curve using a Fourier 
descriptor also is possible. For a detailed 

10 explanation, see "Image Processing Engineering; 

Introductory Edition" edited by Yoshiharu Taniguchi. 
Furthermore, in this embodiment, any parameter may be 
selected for the comparison of shape data and there is 
no limitation whatsoever upon the comparison method. 

15 In other words, this embodiment is characterized in 
that the method of comparing shape data may be a 
comparison of any of the area, perimeter, wavelet 
identifier, circularity, centroid or moment of the 
shape data, or a combination thereof. 

20 Processing branches depending upon the result of 

comparison (step S203). If the result of the 
comparison is that the difference is small ("NO" at 
step S203), it is determined that there is no 
abnormality. If the result of the comparison is that 

25 the difference is large ("YES" at step S203), then the 
shape data is examined. For example, if we let 
reference numerals 1401 and 1402 represent shape data 
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of the preceding frame and shape data of the present 
frame, respectively, then the difference between the 
frames will be large and, hence, the shape data 1402 
is examined to determine whether this frame is an 
5 abnormal frame. 

As for the examination procedure, first the shape 
data of the succeeding frame is generated (step S204). 
Next, the shape data of the preceding frame is 
compared with the shape data of the succeeding frame 

10 (step S205). The method of comparison in this case 
may be the same as that carried out at step S202. 
Processing branches depending upon the result of this 
comparison (step S206). 

If the decision rendered at step S206 is that the 

15 difference is small ("YES" at step S206), then it is 
judged that the shape data of the present frame is 
abnormal. If the difference is large ("NO" at step 
S206), on the other hand, this means that the change 
is not a momentary change and therefore it is judged 

20 that a flash is not the cause of the change. In Fig. 
15, since the difference between the data 1401 of the 
preceding frame and data 1403 of the succeeding frame 
is small, it is judged that frame 1402 is one in which 
the flash was fired (i.e., that this is an abnormal 

25 frame) . If an abnormality is determined, the data of 
the present frame (the shape data and image data of 
the present frame) is corrected (generated again) 
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(step S207). 

More specifically, the image processing apparatus 
according to this embodiment is characterized in that 
the abnormal - data discrimination unit 105 
5 discriminates the necessity of the shape data 

correction by comparing the shape data of an object 
from frame to frame of a plurality of frames that 
differ in time. 

Further, the image processing apparatus according 

10 to this embodiment is characterized in that the 

abnormal-data discrimination unit 105 compares the 
shape data of the object using the present frame and 
the preceding frame (step S202), compares the shape 
data of the object using the preceding frame and the 

15 succeeding frame (step S205), and decides that 

correction of the present frame is necessary if the 
difference in the shape data of the object between the 
present frame and the preceding frame is large and, 
moreover, the difference in the shape data of the 

20 object between the preceding frame and the succeeding 
frame is small. 

The correction processing executed at step S207 
will be described in greater detail later. 

Fig. 4 is a flowchart useful in describing 

25 another example of discrimination processing for 

detecting a frame, in which a flash has been fired, in 
the abnormal -data discrimination unit 105 according to 



- 21 - 



the first embodiment. In the flowchart of Fig. 4, 
processing steps other than steps S208, S209 are 
processing steps identical with those shown in Fig. 3. 
The shape data of the present frame is compared 
5 with the shape data of the succeeding frame at step 
S205. Further, if the decision rendered at step S209 
is that the difference is large, this means that a 
flash has been fired; if the difference is small, this 
means that the flash has not been fired. If the 

10 difference is small, it can be considered that the 
state of the object has undergone a transition and 
that the object has attained a new state. For 
example, in Fig. 15, the difference between the shape 
data 1402 of the present frame and the shape data 1403 

15 of the succeeding frame is large, it is judged that 
frame 1402 is one in which the flash was fired. 

Further, in a special case in which a sudden 
change in shape data cannot be conceived as being 
ascribable to a cause other than a flash, it is 

20 possible to proceed to step S207 based solely on the 
result of the decision at step S203 without using the 
data of the succeeding frame. 

In other words, the image processing apparatus 
according to this embodiment is characterized in that 

25 the abnormal-data discrimination unit 105 compares the 
shape data of the object using a present frame and a 
preceding frame (step S202), compares the shape data 
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of the object using the present frame and the 
succeeding frame (step S208). and decides that 
correction of the present frame is necessary if the 
difference in the shape data of the object between the 
5 present frame and the preceding frame is large and, 
moreover, the difference in the shape data of the 
object between the present frame and the succeeding 

frame is small . 

The processing (step S207) of a frame judged to 
10 require correction will now be described. 

Fig. 5 is a flowchart useful in describing 
processing for correcting shape data and image data in 
the first embodiment of the invention. If an object 
to be extracted is a rigid body, the entire body will 
15 move as one. If this is not the case, however, 

portions that exhibit different motion will appear 
locally. In the description that follows, an area in 
which all movements coincide shall be referred to as a 
"common area", and areas other than these in which 
20 local motion occurs shall be referred to as "motion 
areas " . 

First, the shape-data correction unit 106 detects 
a common area from shape data of the preceding and 
succeeding frames (step S301). This can be found in 
25 simple fashion by performing an AND operation between 
the shape data of the preceding frame and the shape 
data of the succeeding frame. In Fig. 7, reference 
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numerals 501 denote shape data of a preceding frame, 

502 shape data of a succeeding frame, and 503 a common 
area detected from these two frames. 

Further, the image-data correction unit 107 
5 generates a common area of image data in the present 
frame (step S302). This is generated by finding the 
average value between the image data of the preceding 
frame and the image data of the succeeding frame. 
That is, let Pp(x,y) and Pb(x,y) represent a pixel 

10 value of a certain point (x,y) in the preceding frame 
and a pixel value of the point in the succeeding 
frame, respectively. The average value can then be 
found by the calculation [Pp(x,y) + Pb(x,y)]/2. 

Meanwhile, the shape-data correction unit 106 

15 detects a motion area of the shape data in the 

preceding frame (step S303). This can be found by 
performing an exclusive-OR operation between the shape 
data 501 of the preceding frame and the common areas 

503 of the preceding and succeeding frames. In Fig. 
20 7, reference numeral 504 denotes the detected motion 

area of the preceding frame. 

The image-data correction unit 107 detects a 
motion area of the image data in the preceding frame 
(step S304). This may be achieved merely by 
25 correlating the image data with the position of the 
shape data of the extracted motion area. The shape- 
data correction unit 106 similarly obtains a motion 
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area of the shape data in the succeeding frame (step 
S305). in this case also the motion area can be found 
by performing an exclusive-OR operation between the 
shape data 502 of the succeeding frame and the common 

5 areas 503 of the preceding and succeeding frames. In 
Fig. 7, reference numeral 505 denotes the result of 
detection. The image-data correction unit 107 then 
detects a motion area of the image data in the 
succeeding frame (step S306). This also may be 

10 achieved merely by correlating the image data with the 
position of the shape data of the extracted motion 
area. 

Next, the shape-data correction unit 106 detects 
movement of the motion area of shape data between the 
15 preceding and succeeding frames (step S307). Pattern 
matching using an af fine transformation or the like 
can be employed in detecting corresponding points. An 
example of an affine transformation is as follows: 
X = (x-xO)cose - (y-yO)sine + x0 
20 y = (x-xO)sine + (y-y0)cos9 + y0 ... (2) 

where (xO.yO) represents the center of rotation and 0 
the angle of rotation. Pattern matching adopts the 
sum total of the differences between the values 
obtained by this calculation and the actual values or 
25 the sum of the squares of these differences as an 
evaluation value and obtains the center position of 
rotation and the angle of rotation that will minimize 
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this value. 

Fig. 8 is a diagram useful in describing result 
of detecting corresponding points in shape data 
according to the first embodiment. In order to 
5 simplify the description, the shape will be described 
as a line segment. In Fig. 8, let line segments 601 
and 603 represent data in the preceding and succeeding 
frames, respectively. In this case, the angle defined 
by the line segments 601 and 603 is the angle of 

10 rotation in the affine transformation, and the 

position of the point at which AO and A2 overlap is 
the center of rotation. Furthermore, points 
corresponding to B0 and CO are B2 and C2, 
respectively. 

15 Next, the shape-data correction unit 106 

calculates motion of the shape data of the present 
frame (step S308). In a case where motion is obtained 
at the rotation angle and center of rotation in the 
manner shown in Fig. 8, the amount of motion can be 

20 found by making the rotation angle and center of 
rotation 0/2 and AO, respectively. 

Furthermore, the shape-data correction unit 106 
generates a motion area of shape data in the present 
frame (step S309). That is, the corresponding points 

25 are found by substituting the parameter calculated at 
step S308. As a result, a point corresponding to AO 
and A2 becomes Al, a point corresponding to B0 and B2 
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becomes Bl. and a point corresponding to CO and C2 
becomes CI. A line segment 602 obtained by connecting 
all of the corresponding points constitutes shape data 
of the present frame. Reference numeral 506 in Fig. 7 
5 denotes shape data of the motion area of the present 
frame obtained in this fashion. 

More specifically, this embodiment is 
characterized in that the shape-data correction unit 
106 corrects the shape data of the object in an Nth 
10 frame, which has been determined to be abnormal, using 
an (N-l)th or earlier frame determined to be normal 
and an (N+l)th or later frame determined to be normal. 

Further, this embodiment is characterized in that 
the shape-data correction unit 106 detects 
15 corresponding points in the shape data between an (N- 
l)th or earlier frame and an (N+l)th or later frame. 
Furthermore, this embodiment is characterized in that 
the shape-data correction unit 106 obtains amount of 
motion of corresponding points by performing pattern 
20 matching with respect to one or each of a plurality of 
prescribed areas. Furthermore, this embodiment is 
characterized in that the amount of motion of the 
corresponding points is amount of movement and amount 
of rotation of the positions of the corresponding 
25 points obtained by an affine transformation. 

Next, the image-data correction unit 107 
generates image data corresponding to the shape data 
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generated at step S309 (step S310). That is, 
utilizing the parameter obtained at step S308, the 
image-data correction unit 107 obtains pixel values 
from the average values of corresponding points, as by 
5 obtaining the pixel value of Al from the average value 
of AO and A2 and the pixel value of Bl from the 
average value of BO and B2. It should be noted that 
this method of obtaining average values is similar to 
that of the procedure described at step S302. 

10 The shape-data correction unit 106 then combines 

the shape data of the common area obtained at step 
S301 and the shape data of the motion area obtained at 
step S309 and re-generates the shape data of the 
present frame (step S311). Reference numeral 507 in 

15 Fig. 7 denotes shape data of the present frame thus 

obtained. Furthermore, the image-data correction unit 
107 combines the image data of the common area 
obtained at step S302 and the image data of the motion 
area obtained at step S310 and re-generates the image 

20 data of the present frame (step S312). 

When the shape data and image data of the present 
frame is re-generated through the procedure described 
above, processing is completed. This data is then 
input to the encoder 104 and subjected to MPEG- 4 

25 coding in the manner described above. 

Thus, in accordance with the image processing 
apparatus according to the first embodiment, as set 
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forth above, a frame in which shape data is 
discontinuous, as in the case of an image illuminated 
by a flash, is detected, and this frame is corrected 
based upon the frames before and after it. As a 
5 result, the occurrence of excessive amount of code can 
be suppressed with regard to both the shape data and 
image data and it is possible to realize a coding 
system that provides excellent visual results, 
< Second Embodiment > 

10 A second embodiment of the present invention will 

be described with reference to Figs. 9 to 12. This 
embodiment relates to processing for dealing with the 
problem (instances where one object that is not to be 
extracted crosses in front of another object that is 

15 to be extracted) described above with reference to 
Fig. 16. 

Fig. 9 is a flowchart for describing operation of 
the abnormal-data discrimination unit 105 according to 
a second embodiment of the present invention. The 

20 processing of steps S701 to S703 is the same as that 
of steps S201 to S203 in Fig. 3 described in 
connection with the first embodiment. Here, however, 
the present frame is described as the Nth frame. 
Further, reference numerals 1501 to 1504 in Fig. 16 

25 shall denote shape data of (N-l)th, Nth, (N+l)th and 
(N+2)th frames, respectively. In this example, the 
shape data 1502 of the present frame is compared with 
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the shape data of the preceding frame and the 
difference is large ("YES" at step S703). The shape 
data, therefore, is examined. 

If an object that is not to be extracted crosses 
5 in front of an object that is to be extracted, it is 
expected that a large change in shape will be detected 
over a plurality of frames. The reason for this is 
that such a change does not appear in only one frame. 
Accordingly, a counter m is reset to 1 (step S704) as 
10 initialization processing for counting up the number 
of frames. 

Next, the abnormal-data discrimination unit 105 
generates shape data of the (N+l)th frame (the frame 
that follows the present frame) (step S705). The 

15 abnormal-data discrimination unit 105 then compares 
the generated shape data of the (N+l)th frame and the 
shape data of the (N-l)th frame (step S706). The 
method of comparison is the same as that of step S703. 
The abnormal-data discrimination unit 105 then 

20 determines whether the difference between the items of 
data is small or not (step S707). If the decision 
rendered is that the difference is small ( "YES" at 
step S707), then it is judged that the shape data of 
the present frame is abnormal. If the difference is 

25 large ("NO" at step S707), on the other hand, then it 
is judged that the next frame also is to be examined. 
In the example shown in Fig. 16, the shape data 1501 
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is compared with the shape data 1503 and the 
difference is judged to be large. 

The abnormal -data discrimination unit 105 
determines whether the counter value m has attained a 
5 maximum value max (step S708). To decide the maximum 
value max, maximum time believed to be necessary for 
an object to cross in front of another object that is 
not to be extracted is set beforehand. For example, 
if the frame rate is 15 fps and a maximum of two 

10 seconds is considered, then max is set to 30. If the 
counter value m has not reached the maximum value m 
("NO" at step S708), then the counter value m is 
incremented (step S709). 

If it is found at step S708 that the counter 

15 value m has reached the maximum value max ("YES" at 
step S708), then error processing is executed (step 
S710). In this case, it is considered not that 
something has crossed in front of the object to be 
extracted but that the shape of this object itself has 

20 changed in a major way. Further, in a case where 

shape data that is not corrected is used as is without 
any problems, no particular error processing is 
executed and the processing of this frame may be 
terminated. 

25 The abnormal-data discrimination unit 105 repeats 

the processing of steps S705 to S709 above and exits 
this loop if it determines at step S707 that the 
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difference is small ("YES" at step S707). The counter 
value m prevailing at this time will be the number of 
frames in which the result of extraction has been 
deemed to be abnormal. This is the number of frames 
5 that require re-generation of data. In the example of 
Fig. 16, the shape data of frame 1501 is compared with 
the shape data of frame 1504 and the loop is exited at 
the moment m becomes 2. 

At steps S711 to S714. processing for correcting 
10 (re-generating) data in the detected m-number of 

frames is executed. First, initialization processing 
for setting a frame-number counter k to zero is 
executed (step S711). Step S712 is the main part of 
correction processing and the details thereof will be 
15 described later. Until end is discriminated (step 
S713) and k becomes equal to m-1, the value of k is 
incremented at step S714 and the above processing is 
repeated. When data re-generation from the Nth frame 
to the (N+m-l)th frame ends, one series of processing 
20 steps ends. In the example of Fig. 16, m is equal to 
2 and the data of frames 1502 and 1503 is re- 
generated . 

In the flowchart of Fig. 9, an example in which 
the Nth frame is processed is described. However, in 
25 a case where data correction has been carried out, the 
frame processed next may be made the (N + m + l)th frame 
rather than the (N+l)th frame. 
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In other words, the image processing apparatus 
according to this embodiment is characterized in that 
the abnormal-data discrimination unit 105 compares the 
shape data of the object using a present frame and a 
5 preceding frame (step S702), compares the shape data 
of the object using frames following the present frame 
and the preceding frame (step S706), and decides that 
correction of a prescribed frame from the present 
frame or later is necessary if the difference between 

10 the present frame and the preceding frame is large 
and, moreover, a difference in the shape data of the 
object between the preceding frame and the frames 
following the present frame is small. 

Processing for correcting a frame judged to 

15 require correction will now be described. 

Fig. 10 is another flowchart useful in describing 
processing for correcting shape data and image data 
according to the second embodiment. 

First, the shape-data correction unit 106 detects 

20 a common area from shape data of the (N-l)th and 
(N+m)th frames (step S801). This can be found in 
simple fashion by performing an AND operation between 
the shape data of the (N-l)th frame and the shape data 
of the (N+m)th frame in a manner similar to that 

25 described above with reference to Fig. 5. Unlike the 
case of Fig. 5, however, it is considered that the 
common area has moved if the time interval between 
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frames is long. In this case, it is so arranged that 
motion of the overall object to be extracted is found 
by pattern matching. If the affine transformation 
described in Fig. 5 is used, the amount of 
5 translational motion and amount of rotation of the 
object between frames can be obtained. 

Specifically, if we let 9 represent the angle of 
rotation from the (N-l)th frame to the (N+m)th frame 
and let (XO,yO) represent the amount of movement 

10 between these frames, then the angle of rotation in 
the (N+k)th frame will be 9 x (k+l)/(m+l) and the 
amount of movement will be [xO x (k+l)/(m+l), yO x 
(k+l)/(m+l) ] . After the positions are made to 
coincide by these parameters, the common area can be 

15 detected by performing the AND operation. 

Fig. 11 is a diagram useful in describing shape 
data according to the second embodiment. In the 
example of Fig. 11, reference numerals 1001 and 1002 
denote shape data in (N-l)th and (N+2)th frames, 

20 respectively. Since there is no movement of the 
common area, the angle of rotation is zero and the 
amount of movement is (0,0). Reference numeral 103 
denotes a common area obtained from the shape data 
1001 and 1002. 

25 If there is local motion outside of the common 

area, then detection of the motion area is performed. 
First, the motion area of the shape data in the (N- 
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l)th frame is detected (step S802). This can be found 
by performing an exclusive-OR operation after the 
positions of the shape data 1001 of the (N-l)th frame 
and common area 1003 are made to coincide- Reference 
5 numeral 1004 denotes the extracted motion area of the 
(N-l)th frame. Next, the motion area of the image 
data in the (N-l)th frame is detected (step S803). 
This is image data that corresponds to the area found 
at step S802. 

10 Similarly, the motion area of the shape data in 

the (N+m)th frame is detected (step S804). This can 
be found by performing an exclusive-OR operation after 
the positions of the shape data 1002 of the (N+m)th 
frame and common area 1003 are made to coincide. 

15 Reference numeral 1005 denotes the detected motion 
area of the (N-l)th frame. Next, the motion area of 
the image data in the (N+m)th frame is detected (step 
S805). This is image data that corresponds to the 
area found at step S804. 

20 Next, movement of the motion area of the shape 

data between the (N-l)th frame and the (N+m)th frame 
is detected (step S806). This involves detecting 
corresponding points between the shape data 1004 of 
the (N-l)th frame and the shape data 1005 of the 

25 (N+m)th frame. Detection of the corresponding points 
employs pattern matching using an affine 
transformation or the like. 
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Fig. 12 is a diagram useful in describing 
processing for detecting corresponding points in shape 
data in the second embodiment. In order to simplify 
the description, the shape will be described as a line 
5 segment. In Fig. 12, let reference numeral 1101 

denote the (N-l)th frame and let a line segment 1104 
represent data in the (N+m)th frame. The angle 
defined by the line segments 1101 and 1104 is an angle 
of rotation 9' in this affine transformation, and the 

10 position of the point at which AO and A3 overlap is 
the center (x'O^'O) of rotation. The points 
corresponding to B0 and CO are B3 and C3, 
respectively. 

The processing of each frame to be corrected is 

15 executed in the steps that follow. First, 

initialization processing is executed (step S807). 
Here the counter k for counting the number of frames 
to be corrected is set to zero. Next, motion of the 
common area in the (N+k)th frame is calculated (step 

20 S808). This is calculated from the amount of movement 
obtained at step S801. In the example of Fig. 11, the 
angle of rotation is zero and the amount of movement 
is (0,0). The shape data of the common area is 
generated from the values obtained at step S808 (step 

25 S809). In the example of Fig. 11, the amount of 

movement of the common area is zero and therefore the 
common area of the shape data in the Nth frame 
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coincides with the common area 1003. 

Next, image data corresponding to the shape data 
obtained at step S809 is generated (step S810). Let 
P0(x,y) and Pm(x,y) represent a pixel value of a 
5 certain point (x f y) in the (N-l)th frame and a pixel 
value of the point in the (N+m)th frame, respectively. 
The pixel value in the (N+k)th frame can then be found 
by the calculation [P0(x,y) x (m-k) + Pm(x,y) x 
(k+1) ]/(m+l) . In a case where it is considered that 

10 there is almost no change in the common area, a simple 
method that may be adopted is to use the average value 
[P0(x,y) + Pm(x,y)]/2. An even simpler approach that 
is conceivable is to utilize P0(x,y) as is. This is 
equivalent to copying the image data of the (N-l)th 

15 frame as is. 

Next, movement of the motion area of the (N+k)th 
frame is calculated (step S811). This is found from 
the detection of the previously obtained movement 
between the motion area in the (N-l)th frame and the 

20 motion area in the (N+m)th frame. Since the angle of 
rotation from the (N-l)th frame to the (N+m)th frame 
is 9 1 and the amount of movement is (x'0,y'0), the 
angle of rotation in the (N+k)th frame is 0' x 
(k+l)(m+l) and the amount of movement is [x'O x 

25 (k+l)/(m+l), y'O x (k+1 ) / (m+1 ) ] . When m=2 holds, the 
angle of rotation in the Nth frame is 0'/3 and the 
amount of movement is (x'0/3, y'0/3). 
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Shape data of the motion area is generated from 
the values obtained at step S811 (step S812). In Fig. 
12, line segment 1102 is obtained if points 
corresponding to the rotation angle 0'/3 and amount of 
5 movement (x'0/3, y'0/3) are found with respect to line 
segment 1101. Next, image data corresponding to the 
shape data of the motion area is generated (step 
S813). The method is the same as that used to 
calculate the pixel values in the common area. If we 

10 let P , 0(x' / y') and P'nUx^y') represent a pixel value 
of a certain point (x'^y 1 ) in the (N-l)th frame and a 
pixel value of the point in the (N+m)th frame, 
respectively, then the pixel value in the (N+k)th 
frame will be found by the calculation [P'Ofx^y 1 ) x 

15 (m-k) + P f m(x f ,y f ) x (k+1) ]/(m+l) . 

The shape data of the (N+k)th frame is re- 
generated (step S814). This is achieved by combining 
the shape data of the common area found at step S809 
and the shape data of the motion area found at step 

20 S812. Reference numeral 1008 in Fig. 1 represents 
shape data that has been re -generated in the Nth 
frame. Next, the image data of the (N+k)th frame is 
re-generated (step S815). This is achieved by 
combining the image data of the common area found at 

25 step S810 and the image data of the motion area found 
at step S813. 

The number of frames is discriminated (step 
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S816). If the number of frames is less than m-1 ("NO" 
at step S816), then the value of k is incremented at 
step S817 and the processing of steps S808 onward is 
repeated. If m is equal to 2, then the data of the 
Nth and (N+l)th frames is re-generated and this series 
of processing steps is terminated. In the (N+l)th 
frame, the angle of rotation is 9' x 2/3 and the amount 
of movement is (x'O x 2/3, y'O x 2/3). If the 
corresponding points are found, line segment 1103 in 
Fig. 12 is obtained. Further, reference numeral 1007 
in Fig. 11 denotes the motion area of the shape data 
in the (N+l)th frame, and reference numeral 1009 
denotes the shape area that has been regenerated in 
the (N+l)th frame. 

Here only one type of local motion outside of the 
motion area has been described. However, if a 
plurality of motions exist, it will suffice to repeat 
processing for motion-area detection a number of times 
equal to the number of motions. 

Thus, in accordance with the second embodiment as 
described above, a plurality of frames in which shape 
data is discontinuous, as in the case of an image 
obtained when an object that is not to be extracted 
crosses in front of an object that is to be extracted, 
are detected, and these frames are corrected based 
upon the frames before and after them. As a result, 
the occurrence of excessive amount of code can be 
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suppressed with regard to both the shape data and 
image data and it is possible to realize a coding 
system that provides excellent visual results. 
< Third Embodiment > 
5 Fig. 6 is another flowchart relating to 

correction of shape data and image data in a third 
embodiment of the present invention. Specifically, 
according to this embodiment, processing for 
correcting the shape data and image data by the shape - 
10 data correction unit 106 and image-data correction 

unit 107 in the first and second embodiments set forth 
above is made processing for replacing this data with 
shape data and image data of a frame determined to be 
normal (i.e., by copying the shape data and image data 
15 of the normal frame) . 

More specifically, this embodiment is 
characterized in that the shape-data correction unit 
106 replaces shape data in an Nth frame determined to 
be abnormal with shape data in an (N-l)th or earlier 
20 frame determined to be normal (step S401). Further, 
this embodiment is characterized in that the image - 
data correction unit 107 generates an Nth frame 
determined to be abnormal from an (N-l)th frame 
determined to be normal and an (N+l)th frame 
25 determined to be normal. Furthermore, the embodiment 
is characterized in that the image -data correction 
unit 107 replaces an Nth frame determined to be 
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abnormal with an (N-l)th frame determined to be normal 
(step S402) . 

< Other Embodiment s> 

The present invention can be applied to a system 
5 constituted by a plurality of devices (e.g., a host 
computer, interface, reader, printer, etc.) or to an 
apparatus comprising a single device (e.g., a copier 
or facsimile machine, etc.). 

Further, it goes without saying that the object 

10 of the invention is attained also by supplying a 
recording medium (or storage medium) on which the 
program codes of the software for performing the 
functions of the foregoing embodiments to a system or 
an apparatus have been recorded, reading the program 

15 codes with a computer (e.g., a CPU or MPU) of the 
system or apparatus from the recording medium, and 
then executing the program codes. In this case, the 
program codes read from the recording medium 
themselves implement the novel functions of the 

20 embodiments, and the program codes per se and 

recording medium storing the program codes constitute 
the invention. Further, besides the case where the 
aforesaid functions according to the embodiments are 
implemented by executing the program codes read by a 

25 computer, it goes without saying that the present 

invention covers a case where an operating system or 
the like running on the computer performs a part of or 
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the entire process based upon the designation of 
program codes and implements the functions according 
to the embodiments. 

It goes without saying that the present invention 
further covers a case where, after the program codes 
read from the recording medium are written in a 
function expansion card inserted into the computer or 
in a memory provided in a function expansion unit 
connected to the computer, a CPU or the like contained 
in the function expansion card or function expansion 
unit performs a part of or the entire process based 
upon the designation of program codes and implements 
the function of the above embodiment. 

In a case where the present invention is applied 
to the above-mentioned recording medium, program code 
corresponding to the flowcharts described earlier is 
stored on the recording medium. 

Thus, in accordance with the present invention, 
as described above, even if shape data representing 
the shape of an object in a moving image undergoes a 
major change temporarily, a frame that is to be 
corrected can be detected and corrected in ideal 
fashion and it is possible to execute moving-image 
coding that is outstanding both visually and in terms 
of coding efficiency. 

The present invention is not limited to the above 
embodiments and various changes and modifications can 
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be made within the spirit and scope of the present 
invention. Therefore, to apprise the public of the 
scope of the present invention, the following claims 
are made. 
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